On the variational distance of independently repeated experiments

Let P and Q be two probability distributions which differ only for values with probability at least p > 0. We show that the variational distance δ(Pn, Qn) between the n-fold product distributions Pn and Qn is upper bounded by √ n/(2p) δ(P,Q), i.e., it cannot grow faster than the square root of n. 1 Preliminaries Let P be a probability distribution with range Z and let n ∈ N. We denote by Pn the n-fold product distribution, that is, P(z1, . . . , zn) = n ∏ i=1 P (zi) for any z1, . . . , zn ∈ Z. Note that Pn describes n independently repeated random experiments with distribution P . The variational distance between two probability distributions P and Q with range Z is defined as1 δ(P,Q) := 1 2 ∑ z∈Z |P (z)−Q(z)| . Note that δ is a distance measure on the set of probability distributions with range Z. In particular, δ is symmetric, δ(P,Q) = 0 if and only if P = Q, and the triangle inequality δ(P,Q) ≤ δ(P,P ) + δ(P , Q) (1) See, e.g., [1]. δ(·, ·) is also called statistical difference [4], Kolmogorov distance, or trace distance [3]. 1 holds. 2 Main Result and Proof 2.1 Upper Bounds for the Variational Distance Let P and Q be two probability distributions with range Z. It is a direct consequence of the triangle inequality (1) that the variational distance δ(Pn, Qn) between the n-fold product distributions Pn and Qn cannot grow faster than linearly in n, i.e., δ(P, Q) ≤ nδ(P,Q) . (2) Moreover, it is easy to find examples where this inequality is almost tight. Let, e.g., P and Q be two binary distributions with range Z = {0, 1} such that P (1) = 0 and Q(1) = ε for some ε > 0. If nε ≪ 1 then the variational distance δ(Pn, Qn) is roughly equal to nδ(P,Q) = nε. However, the upper bound (2) can only be close to optimal if, for some element z ∈ Z, the relative difference |P (z)−Q(z)|/(P (z) +Q(z)) between the probabilities is large. (Note that, in the above example, this relative difference for z = 1 equals one.) Indeed, the following result states that, in all other cases, δ(Pn, Qn) cannot grow more than the square root of n. Lemma 2.1. Let P and Q be two probability distributions with range Z, let D := {z ∈ Z : P (z) 6= Q(z)} be the subset of Z where P and Q differ, and let p̄ := infz∈D(min(P (z), Q(z))). If p̄ > 0 then, for any n ∈ N, δ(P, Q) ≤ √